Cumulated Relative Position: A Metric for Ranking Evaluation

Measuring is a key to scientific progress. This is particularly true for research concerning complex systems. Multilingual and multimedia information access systems, such as search engines, are increasingly complex: they need to satisfy diverse user needs and support challenging tasks. Their development calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness. Information Retrieval (IR) research offers a strong evaluation methodology and a range of evaluation metrics, such as Mean Average Precision (MAP) and Normalized Discounted Cumulated Gain ((n)DCG). In this paper, we propose a new metric for ranking evaluation, the Cumulated Relative Position (CRP). We start with the observation that a document of a given degree of relevance may be ranked too early or too late regarding the ideal ranking of documents for a query. Its relative position may be negative, indicating too early ranking, zero indicating correct ranking, or positive, indicating too late ranking. By cumulating these relative rankings we indicate, at each ranked position, the net effect of document displacements, the CRP. We first define the metric formally and then discuss its properties, its relationship to prior metrics, and its visualization. Finally we propose different visualizations of CRP by exploiting a test collection to demonstrate its behavior.